{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-end Reusable ML Pipeline with Seldon and Kubeflow\n",
    "\n",
    "In this example we showcase how to build re-usable components to build an ML pipeline that can be trained and deployed at scale.\n",
    "\n",
    "We will automate content moderation on the Reddit comments in /r/science building a machine learning NLP model with the following components:\n",
    "\n",
    "![](img/completed-pipeline-deploy.jpg)\n",
    "\n",
    "This tutorial will break down in the following sections:\n",
    "\n",
    "1) Run all the services (Kubeflow and Seldon)\n",
    "\n",
    "2) Test and build all our reusable pipeline steps\n",
    "\n",
    "3) Use Kubeflow to Train the Pipeline and Deploy to Seldon\n",
    "\n",
    "5) Test Seldon Deployed ML REST Endpoints\n",
    "\n",
    "6) Visualise Seldon's Production ML Pipelines\n",
    "\n",
    "## Before you start\n",
    "Make sure you install the following dependencies, as they are critical for this example to work:\n",
    "\n",
    "* Helm v2.13.1+\n",
    "* A Kubernetes cluster running v1.13 or above (minkube / docker-for-windows work well if enough RAM)\n",
    "* kubectl v1.14+\n",
    "* ksonnet v0.13.1+\n",
    "* kfctl 0.5.1 - Please use this exact version as there are major changes every few months\n",
    "* Python 3.6+\n",
    "* Python DEV requirements (we'll install them below)\n",
    "\n",
    "Let's get started! 🚀🔥 We will be building the end-to-end pipeline below:\n",
    "\n",
    "![](img/kubeflow-seldon-nlp-full.jpg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python-dateutil\r\n",
      "https://storage.googleapis.com/ml-pipeline/release/0.1.20/kfp.tar.gz\r\n",
      "kubernetes\r\n",
      "click\r\n",
      "seldon_core\r\n",
      "numpy\r\n"
     ]
    }
   ],
   "source": [
    "!cat requirements-dev.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install -r requirements-dev.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Run all the services (Kubeflow and Seldon)\n",
    "Kubeflow's CLI allows us to create a project which will allow us to build the configuration we need to deploy our kubeflow and seldon clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "app.yaml\r\n"
     ]
    }
   ],
   "source": [
    "!kfctl init kubeflow-seldon\n",
    "!ls kubeflow-seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we run the following commands to basically launch our Kubeflow cluster with all its components. \n",
    "\n",
    "It may take a while to download all the images for Kubeflow so feel free to make yourself a cup of ☕.\n",
    "\n",
    "If you have a terminal you can see how the containers are created in real-time by running `kubectl get pods -n kubeflow -w`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "cd kubeflow-seldon\n",
    "kfctl generate all -V\n",
    "kfctl apply all -V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's run Seldon \n",
    "For this we'll need Helm to be running, so we'll initialise it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$HELM_HOME has been configured at /home/alejandro/.helm.\n",
      "Warning: Tiller is already installed in the cluster.\n",
      "(Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)\n",
      "Happy Helming!\n",
      "deployment \"tiller-deploy\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "helm init \n",
    "kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once it's running we can now run the installation command for Seldon.\n",
    "\n",
    "As you can see, we are running the Seldon Operator in the Kubeflow namespace. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   old-mastiff\n",
      "LAST DEPLOYED: Tue May 28 11:47:35 2019\n",
      "NAMESPACE: kubeflow\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ClusterRole\n",
      "NAME                          AGE\n",
      "seldon-operator-manager-role  0s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME                                 AGE\n",
      "seldon-operator-manager-rolebinding  0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                  READY  STATUS             RESTARTS  AGE\n",
      "seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                                   TYPE    DATA  AGE\n",
      "seldon-operator-webhook-server-secret  Opaque  0     0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                                        TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)  AGE\n",
      "seldon-operator-controller-manager-service  ClusterIP  10.101.228.36  <none>       443/TCP  0s\n",
      "\n",
      "==> v1/StatefulSet\n",
      "NAME                                READY  AGE\n",
      "seldon-operator-controller-manager  0/1    0s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install seldon-core-operator --namespace kubeflow --repo https://storage.googleapis.com/seldon-charts --set usageMetrics.enabled=true "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check all the Seldon Deployment is running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon-operator-controller-manager-0                       1/1     Running   1          6s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pod -n kubeflow | grep seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Temporary fix for Argo image\n",
    "\n",
    "At the time of writing we need to make some updates in the Argo images with the following commands below.\n",
    "\n",
    "(This basically changes the images to the latest ones, otherwise we will get an error when we attach the volume)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment.extensions/workflow-controller patched\n",
      "deployment.extensions/ml-pipeline patched\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kubeflow patch deployments. workflow-controller --patch '{\"spec\": {\"template\": {\"spec\": {\"containers\": [{\"name\": \"workflow-controller\", \"image\": \"argoproj/workflow-controller:v2.3.0-rc3\"}]}}}}'\n",
    "!kubectl -n kubeflow patch deployments. ml-pipeline --patch '{\"spec\": {\"template\": {\"spec\": {\"containers\": [{\"name\": \"ml-pipeline-api-server\", \"image\": \"elikatsis/ml-pipeline-api-server:0.1.18-pick-1289\"}]}}}}'\n",
    "# !kubectl -n kubeflow patch configmaps workflow-controller-configmap --patch '{\"data\": {\"config\": \"{ executorImage: argoproj/argoexec:v2.3.0-rc3,artifactRepository:{s3: {bucket: mlpipeline,keyPrefix: artifacts,endpoint: minio-service.kubeflow:9000,insecure: true,accessKeySecret: {name: mlpipeline-minio-artifact,key: accesskey},secretKeySecret: {name: mlpipeline-minio-artifact,key: secretkey}}}}\" }}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last command you need to run actually needs to be manual as the patch cannot change configmap contents directly\n",
    "\n",
    "You need to run the edit commad and change the executorImage to: `argoproj/argoexec:v2.3.0-rc3`\n",
    "\n",
    "The command should be run from a terminal:\n",
    "\n",
    "```\n",
    "kubectl edit configmaps workflow-controller-configmap -n kubeflow\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Test and build all our reusable pipeline steps\n",
    "\n",
    "We will start by building each of the components in our ML pipeline. \n",
    "\n",
    "![](img/kubeflow-seldon-nlp-reusable-components.jpg)\n",
    "\n",
    "### Let's first have a look at our clean_text step:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clean_text\t lr_text_classifier  tfidf_vectorizer\r\n",
      "data_downloader  spacy_tokenize\r\n"
     ]
    }
   ],
   "source": [
    "!ls pipeline/pipeline_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like in this step, all of the other steps can be found in the `pipeline/pipeline_steps/` folder, and all have the following structure:\n",
    "* `pipeline_step.py` which exposes the functionality through a CLI \n",
    "* `Transformer.py` which transforms the data accordingly\n",
    "* `requirements.txt` which states the python dependencies to run\n",
    "* `build_image.sh` which uses `s2i` to build the image with one line\n",
    "\n",
    "### Let's check out the CLI for clean_text\n",
    "The pipeline_step CLI is the entry point for the kubeflow image as it will be able to pass any relevant parameters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: pipeline_step.py [OPTIONS]\r\n",
      "\r\n",
      "Options:\r\n",
      "  --in-path TEXT\r\n",
      "  --out-path TEXT\r\n",
      "  --help           Show this message and exit.\r\n"
     ]
    }
   ],
   "source": [
    "!python pipeline/pipeline_steps/clean_text/pipeline_step.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is actually a very simple file, as we are using the click library to define the commands:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import dill\r\n",
      "import click\r\n",
      "import dill\r\n",
      "try:\r\n",
      "    # Running for tests\r\n",
      "    from .Transformer import Transformer\r\n",
      "except:\r\n",
      "    # Running from CLI\r\n",
      "    from Transformer import Transformer\r\n",
      "\r\n",
      "@click.command()\r\n",
      "@click.option('--in-path', default=\"/mnt/raw_text.data\")\r\n",
      "@click.option('--out-path', default=\"/mnt/clean_text.data\")\r\n",
      "def run_pipeline(in_path, out_path):\r\n",
      "    clean_text_transformer = Transformer()\r\n",
      "    with open(in_path, 'rb') as in_f:\r\n",
      "        x = dill.load(in_f)\r\n",
      "    y = clean_text_transformer.predict(x)\r\n",
      "    with open(out_path, \"wb\") as out_f:\r\n",
      "        dill.dump(y, out_f)\r\n",
      "\r\n",
      "if __name__ == \"__main__\":\r\n",
      "    run_pipeline()\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat pipeline/pipeline_steps/clean_text/pipeline_step.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Transformer is where the data munging and transformation stage comes in, which will be wrapped by the container and exposed through the Seldon Engine to ensure our pipeline can be used in production.\n",
    "\n",
    "Seldon provides multiple different features, such as abilities to send custom metrics, pre-process / post-process data and more. In this example we will only be exposing the `predict` step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import re \r\n",
      "from html.parser import HTMLParser\r\n",
      "import numpy as np\r\n",
      "import logging\r\n",
      "\r\n",
      "class Transformer():\r\n",
      "    __html_parser = HTMLParser()\r\n",
      "    __uplus_pattern = \\\r\n",
      "        re.compile(\"\\<[uU]\\+(?P<digit>[a-zA-Z0-9]+)\\>\")\r\n",
      "    __markup_link_pattern = \\\r\n",
      "        re.compile(\"\\[(.*)\\]\\((.*)\\)\")\r\n",
      "\r\n",
      "    def predict(self, X, feature_names=[]):\r\n",
      "        logging.warning(X)\r\n",
      "        f = np.vectorize(Transformer.transform_clean_text)\r\n",
      "        X_clean = f(X)\r\n",
      "        logging.warning(X_clean)\r\n",
      "        return X_clean\r\n",
      "\r\n",
      "    def fit(self, X, y=None, **fit_params):\r\n",
      "        return self\r\n",
      "    \r\n",
      "    @staticmethod\r\n",
      "    def transform_clean_text(raw_text):\r\n",
      "        try:\r\n",
      "            decoded = raw_text.encode(\"ISO-8859-1\").decode(\"utf-8\")\r\n",
      "        except:\r\n",
      "            decoded = raw_text.encode(\"ISO-8859-1\").decode(\"cp1252\")\r\n",
      "        html_unescaped =Transformer.\\\r\n",
      "            __html_parser.unescape(decoded) \r\n",
      "        html_unescaped = re.sub(r\"\\r\\n\", \" \", html_unescaped)\r\n",
      "        html_unescaped = re.sub(r\"\\r\\r\\n\", \" \", html_unescaped)\r\n",
      "        html_unescaped = re.sub(r\"\\r\", \" \", html_unescaped)\r\n",
      "        html_unescaped = html_unescaped.replace(\"&gt;\", \" > \")\r\n",
      "        html_unescaped = html_unescaped.replace(\"&lt;\", \" < \")\r\n",
      "        html_unescaped = html_unescaped.replace(\"--\", \" - \")\r\n",
      "        html_unescaped = Transformer.__uplus_pattern.sub(\r\n",
      "            \" U\\g<digit> \", html_unescaped)\r\n",
      "        html_unescaped = Transformer.__markup_link_pattern.sub(\r\n",
      "            \" \\1 \\2 \", html_unescaped)\r\n",
      "        html_unescaped = html_unescaped.replace(\"\\\\\", \"\")\r\n",
      "        return html_unescaped\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat pipeline/pipeline_steps/clean_text/Transformer.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to understand how the CLI pipeline talks to each other, have a look at the end to end test in `pipeline/pipeline_tests/`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m============================= test session starts ==============================\u001b[0m\n",
      "platform linux -- Python 3.7.3, pytest-4.5.0, py-1.8.0, pluggy-0.11.0\n",
      "rootdir: /home/alejandro/Programming/kubernetes/seldon/seldon-core/examples/kubeflow\n",
      "collected 1 item                                                               \u001b[0m\u001b[1m\n",
      "\n",
      "pipeline/pipeline_tests/test_pipeline.py \u001b[32m.\u001b[0m\u001b[36m                               [100%]\u001b[0m\n",
      "\n",
      "\u001b[33m\u001b[1m===================== 1 passed, 9 warnings in 3.26 seconds =====================\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pytest ./pipeline/pipeline_tests/. --disable-pytest-warnings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build the image we provide a build script in each of the steps that contains the instructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash\r\n",
      "\r\n",
      "s2i build . seldonio/seldon-core-s2i-python3:0.6 clean_text_transformer:0.1\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat pipeline/pipeline_steps/clean_text/build_image.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only thing you need to make sure is that Seldon knows how to wrap the right model and file.\n",
    "\n",
    "This can be achieved with the s2i/environment file. \n",
    "\n",
    "As you can see, here we just tell it we want it to use our `Transformer.py` file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MODEL_NAME=Transformer\r\n",
      "API_TYPE=REST\r\n",
      "SERVICE_TYPE=MODEL\r\n",
      "PERSISTENCE=0\r\n"
     ]
    }
   ],
   "source": [
    "!cat pipeline/pipeline_steps/clean_text/.s2i/environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this is defined, the only thing we need to do is to run the `build_image.sh` for all the reusable components.\n",
    "\n",
    "Here we show the manual way to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# we must be in the same directory\n",
    "cd pipeline/pipeline_steps/clean_text/ && ./build_image.sh\n",
    "cd ../data_downloader && ./build_image.sh\n",
    "cd ../lr_text_classifier && ./build_image.sh\n",
    "cd ../spacy_tokenize && ./build_image.sh\n",
    "cd ../tfidf_vectorizer && ./build_image.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Train our NLP Pipeline through the Kubeflow UI\n",
    "We can access the Kubeflow dashboard to train our ML pipeline via http://localhost/_/pipeline-dashboard\n",
    "\n",
    "If you can't edit this, you need to make sure that the ambassador gateway service is accessible:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE\r\n",
      "ambassador   NodePort   10.97.236.196   <none>        80:30209/TCP   8m58s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get svc ambassador -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my case, I need to change the kind from `NodePort` into `LoadBalancer` which can be done with the following command:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service/ambassador patched\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl patch svc ambassador --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"LoadBalancer\"}]' -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that I've changed it to a loadbalancer, it has allocated the external IP as my localhost so I can access it at http://localhost/_/pipeline-dashboard\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE\r\n",
      "ambassador   LoadBalancer   10.97.236.196   localhost     80:30209/TCP   9m20s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get svc ambassador -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this was successfull, you should be able to access the dashboard\n",
    "![](img/k-pipeline-dashboard.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the pipeline\n",
    "Now we want to generate the pipeline. For this we can use the DSL provided by kubeflow to define the actual steps required. \n",
    "\n",
    "The pipeline will look as follows:\n",
    "\n",
    "![](img/kubeflow-seldon-nlp-ml-pipelines.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "import kfp.dsl as dsl\r\n",
      "import yaml\r\n",
      "from kubernetes import client as k8s\r\n",
      "\r\n",
      "\r\n",
      "@dsl.pipeline(\r\n",
      "  name='NLP',\r\n",
      "  description='A pipeline demonstrating reproducible steps for NLP'\r\n",
      ")\r\n",
      "def nlp_pipeline(\r\n",
      "        csv_url=\"https://raw.githubusercontent.com/axsauze/reddit-classification-exploration/master/data/reddit_train.csv\",\r\n",
      "        csv_encoding=\"ISO-8859-1\",\r\n",
      "        features_column=\"BODY\",\r\n",
      "        labels_column=\"REMOVED\",\r\n",
      "        raw_text_path='/mnt/text.data',\r\n",
      "        labels_path='/mnt/labels.data',\r\n",
      "        clean_text_path='/mnt/clean.data',\r\n",
      "        spacy_tokens_path='/mnt/tokens.data',\r\n",
      "        tfidf_vectors_path='/mnt/tfidf.data',\r\n",
      "        lr_prediction_path='/mnt/prediction.data',\r\n",
      "        tfidf_model_path='/mnt/tfidf.model',\r\n",
      "        lr_model_path='/mnt/lr.model',\r\n",
      "        lr_c_param=0.1,\r\n",
      "        tfidf_max_features=10000,\r\n",
      "        tfidf_ngram_range=3,\r\n",
      "        batch_size='100'):\r\n",
      "    \"\"\"\r\n",
      "    Pipeline \r\n",
      "    \"\"\"\r\n",
      "    vop = dsl.VolumeOp(\r\n",
      "      name='my-pvc',\r\n",
      "      resource_name=\"my-pvc\",\r\n",
      "      modes=[\"ReadWriteMany\"],\r\n",
      "      size=\"1Gi\"\r\n",
      "    )\r\n",
      "\r\n",
      "    download_step = dsl.ContainerOp(\r\n",
      "        name='data_downloader',\r\n",
      "        image='data_downloader:0.1',\r\n",
      "        command=\"python\",\r\n",
      "        arguments=[\r\n",
      "            \"/microservice/pipeline_step.py\",\r\n",
      "            \"--labels-path\", labels_path,\r\n",
      "            \"--features-path\", raw_text_path,\r\n",
      "            \"--csv-url\", csv_url,\r\n",
      "            \"--csv-encoding\", csv_encoding,\r\n",
      "            \"--features-column\", features_column,\r\n",
      "            \"--labels-column\", labels_column\r\n",
      "        ],\r\n",
      "        pvolumes={\"/mnt\": vop.volume}\r\n",
      "    )\r\n",
      "\r\n",
      "    clean_step = dsl.ContainerOp(\r\n",
      "        name='clean_text',\r\n",
      "        image='clean_text_transformer:0.1',\r\n",
      "        command=\"python\",\r\n",
      "        arguments=[\r\n",
      "            \"/microservice/pipeline_step.py\",\r\n",
      "            \"--in-path\", raw_text_path,\r\n",
      "            \"--out-path\", clean_text_path,\r\n",
      "        ],\r\n",
      "        pvolumes={\"/mnt\": download_step.pvolume}\r\n",
      "    )\r\n",
      "\r\n",
      "    tokenize_step = dsl.ContainerOp(\r\n",
      "        name='tokenize',\r\n",
      "        image='spacy_tokenizer:0.1',\r\n",
      "        command=\"python\",\r\n",
      "        arguments=[\r\n",
      "            \"/microservice/pipeline_step.py\",\r\n",
      "            \"--in-path\", clean_text_path,\r\n",
      "            \"--out-path\", spacy_tokens_path,\r\n",
      "        ],\r\n",
      "        pvolumes={\"/mnt\": clean_step.pvolume}\r\n",
      "    )\r\n",
      "\r\n",
      "    vectorize_step = dsl.ContainerOp(\r\n",
      "        name='vectorize',\r\n",
      "        image='tfidf_vectorizer:0.1',\r\n",
      "        command=\"python\",\r\n",
      "        arguments=[\r\n",
      "            \"/microservice/pipeline_step.py\",\r\n",
      "            \"--in-path\", spacy_tokens_path,\r\n",
      "            \"--out-path\", tfidf_vectors_path,\r\n",
      "            \"--max-features\", tfidf_max_features,\r\n",
      "            \"--ngram-range\", tfidf_ngram_range,\r\n",
      "            \"--action\", \"train\",\r\n",
      "            \"--model-path\", tfidf_model_path,\r\n",
      "        ],\r\n",
      "        pvolumes={\"/mnt\": tokenize_step.pvolume}\r\n",
      "    )\r\n",
      "\r\n",
      "    predict_step = dsl.ContainerOp(\r\n",
      "        name='predictor',\r\n",
      "        image='lr_text_classifier:0.1',\r\n",
      "        command=\"python\",\r\n",
      "        arguments=[\r\n",
      "            \"/microservice/pipeline_step.py\",\r\n",
      "            \"--in-path\", tfidf_vectors_path,\r\n",
      "            \"--labels-path\", labels_path,\r\n",
      "            \"--out-path\", lr_prediction_path,\r\n",
      "            \"--c-param\", lr_c_param,\r\n",
      "            \"--action\", \"train\",\r\n",
      "            \"--model-path\", lr_model_path,\r\n",
      "        ],\r\n",
      "        pvolumes={\"/mnt\": vectorize_step.pvolume}\r\n",
      "    )\r\n",
      "\r\n",
      "    try:\r\n",
      "        seldon_config = yaml.load(open(\"../deploy_pipeline/seldon_production_pipeline.yaml\"))\r\n",
      "    except:\r\n",
      "        # If this file is run from the project core directory \r\n",
      "        seldon_config = yaml.load(open(\"deploy_pipeline/seldon_production_pipeline.yaml\"))\r\n",
      "\r\n",
      "    deploy_step = dsl.ResourceOp(\r\n",
      "        name=\"seldondeploy\",\r\n",
      "        k8s_resource=seldon_config,\r\n",
      "        attribute_outputs={\"name\": \"{.metadata.name}\"})\r\n",
      "\r\n",
      "    deploy_step.after(predict_step)\r\n",
      "\r\n",
      "if __name__ == '__main__':\r\n",
      "  import kfp.compiler as compiler\r\n",
      "  compiler.Compiler().compile(nlp_pipeline, __file__ + '.tar.gz')\r\n"
     ]
    }
   ],
   "source": [
    "!cat train_pipeline/nlp_pipeline.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Breaking down the  code\n",
    "As you can see in the DSL, we have the ContainerOp - each of those is a step in the Kubeflow pipeline.\n",
    "\n",
    "At the end we can see the `seldondeploy` step which basically deploys the trained pipeline\n",
    "\n",
    "The definition of the SeldonDeployment graph is provided in the `deploy_pipeline/seldon_production_pipeline.yaml` file.\n",
    "\n",
    "The seldondeployment file defines our production execution graph using the same reusable components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---\r\n",
      "apiVersion: machinelearning.seldon.io/v1alpha2\r\n",
      "kind: SeldonDeployment\r\n",
      "metadata:\r\n",
      "  labels:\r\n",
      "    app: seldon\r\n",
      "  name: \"seldon-deployment-{{workflow.name}}\"\r\n",
      "  namespace: kubeflow\r\n",
      "spec:\r\n",
      "  annotations:\r\n",
      "    project_name: NLP Pipeline\r\n",
      "    deployment_version: v1\r\n",
      "  name: \"seldon-deployment-{{workflow.name}}\"\r\n",
      "  oauth_key: oauth-key\r\n",
      "  oauth_secret: oauth-secret\r\n",
      "  predictors:\r\n",
      "  - componentSpecs:\r\n",
      "    - spec:\r\n",
      "        containers:\r\n",
      "        - image: clean_text_transformer:0.1\r\n",
      "          imagePullPolicy: IfNotPresent\r\n",
      "          name: cleantext\r\n",
      "          resources:\r\n",
      "            requests:\r\n",
      "              memory: 1Mi\r\n",
      "        - image: spacy_tokenizer:0.1\r\n",
      "          imagePullPolicy: IfNotPresent\r\n",
      "          name: spacytokenizer\r\n",
      "        - image: tfidf_vectorizer:0.1\r\n",
      "          imagePullPolicy: IfNotPresent\r\n",
      "          name: tfidfvectorizer\r\n",
      "          volumeMounts:\r\n",
      "          - name: mypvc\r\n",
      "            mountPath: /mnt\r\n",
      "        - image: lr_text_classifier:0.1\r\n",
      "          imagePullPolicy: IfNotPresent\r\n",
      "          name: lrclassifier\r\n",
      "          volumeMounts:\r\n",
      "          - name: mypvc\r\n",
      "            mountPath: /mnt\r\n",
      "        terminationGracePeriodSeconds: 20\r\n",
      "        volumes:\r\n",
      "        - name: mypvc\r\n",
      "          persistentVolumeClaim:\r\n",
      "            claimName: \"{{workflow.name}}-my-pvc\"\r\n",
      "    graph:\r\n",
      "      children:\r\n",
      "      - name: spacytokenizer\r\n",
      "        endpoint:\r\n",
      "          type: REST\r\n",
      "        type: MODEL\r\n",
      "        children:\r\n",
      "        - name: tfidfvectorizer\r\n",
      "          endpoint:\r\n",
      "            type: REST\r\n",
      "          type: MODEL\r\n",
      "          children:\r\n",
      "          - name: lrclassifier\r\n",
      "            endpoint:\r\n",
      "              type: REST\r\n",
      "            type: MODEL\r\n",
      "            children: []\r\n",
      "      name: cleantext\r\n",
      "      endpoint:\r\n",
      "        type: REST\r\n",
      "      type: MODEL\r\n",
      "    name: single-model\r\n",
      "    replicas: 1\r\n",
      "    annotations:\r\n",
      "      predictor_version: v1\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!cat deploy_pipeline/seldon_production_pipeline.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seldon Production pipeline contents\n",
    "If we look at the file we'll be using to deploy our pipeline, we can see that it has the following key points:\n",
    "\n",
    "1) Reusable components definitions as containerSpecs: cleantext, spacytokenizer, tfidfvectorizer & lrclassifier\n",
    "\n",
    "2) DAG (directed acyclic graph) definition for REST pipeline: cleantext -> spacytokenizer -> tfidfvectorizer -> lrclassifier\n",
    "\n",
    "This graph in our production deployment looks as follows:\n",
    "\n",
    "![](img/kubeflow-seldon-nlp-ml-pipelines-deploy.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the pipeline files to upload to Kubeflow\n",
    "To generate the pipeline we just have to run the pipeline file, which will output the `tar.gz` file that will be uploaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nlp_pipeline.py\n",
      "nlp_pipeline.py.tar.gz\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Generating graph definition\n",
    "python train_pipeline/nlp_pipeline.py\n",
    "ls train_pipeline/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Run the pipeline\n",
    "\n",
    "We now need to upload the resulting `nlp_pipeline.py.tar.gz` file generated.\n",
    "\n",
    "This can be done through the \"Upload PIpeline\" button in the UI at http://localhost/_/pipeline-dashboard.\n",
    "\n",
    "Once it's uploaded, we want to create and trigger a run! You should now be able to see how each step is executed:\n",
    "\n",
    "![](img/running-pipeline.jpg)\n",
    "\n",
    "### Inspecting the data created in the Persistent Volume\n",
    "The pipeline saves the output of the pipeline together with the trained model in the persistent volume claim.\n",
    "\n",
    "The persistent volume claim is the same name as the argo workflow:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME        AGE\r\n",
      "nlp-bddff   2m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get workflow -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our workflow is there! So we can actually access it by running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nlp-bddff"
     ]
    }
   ],
   "source": [
    "!kubectl get workflow -n kubeflow -o jsonpath='{.items[0].metadata.name}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can use good old `sed` to insert this workflow name in our PVC-Access controler which we can use to inspect the contents of the volume:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apiVersion: v1\r\n",
      "kind: Pod\r\n",
      "metadata:\r\n",
      "  name: pvc-access-container\r\n",
      "spec:\r\n",
      "  containers:\r\n",
      "  - name: pvc-access-container\r\n",
      "    image: busybox\r\n",
      "    command: [\"/bin/sh\", \"-ec\", \"sleep 1000\"]\r\n",
      "    volumeMounts:\r\n",
      "    - name: mypvc\r\n",
      "      mountPath: /mnt\r\n",
      "  volumes:\r\n",
      "  - name: mypvc\r\n",
      "    persistentVolumeClaim:\r\n",
      "      claimName: nlp-b7qt8-my-pvc\r\n"
     ]
    }
   ],
   "source": [
    "!sed \"s/PVC_NAME/\"$(kubectl get workflow -n kubeflow -o jsonpath='{.items[0].metadata.name}')\"-my-pvc/g\" deploy_pipeline/pvc-access.yaml "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just need to apply this container with our kubectl command, and we can use it to inspect the mounted folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/pvc-access-container created\r\n"
     ]
    }
   ],
   "source": [
    "!sed \"s/PVC_NAME/\"$(kubectl get workflow -n kubeflow -o jsonpath='{.items[0].metadata.name}')\"-my-pvc/g\" deploy_pipeline/pvc-access.yaml | kubectl -n kubeflow apply -f -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                   READY   STATUS    RESTARTS   AGE\r\n",
      "pvc-access-container   1/1     Running   0          6s\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get pods -n kubeflow pvc-access-container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can run an `ls` command to see what's inside:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;32mclean.data\u001b[m       \u001b[1;32mlr.model\u001b[m         \u001b[1;32mtext.data\u001b[m        \u001b[1;32mtfidf.model\u001b[m\r\n",
      "\u001b[1;32mlabels.data\u001b[m      \u001b[1;32mprediction.data\u001b[m  \u001b[1;32mtfidf.data\u001b[m       \u001b[1;32mtokens.data\u001b[m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kubeflow exec -it pvc-access-container ls /mnt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod \"pvc-access-container\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f deploy_pipeline/pvc-access.yaml -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Test Deployed ML REST Endpoints\n",
    "Now that it's running we have a production ML text pipeline that we can Query using REST and GRPC\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we can check if our Seldon deployment is running with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                          AGE\r\n",
      "seldon-deployment-nlp-b7qt8   57m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kubeflow get seldondeployment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need the Seldon Pipeline Deployment name to reach the API, so we can get it using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldon-deployment-nlp-b7qt8"
     ]
    }
   ],
   "source": [
    "!kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can interact with our API in two ways: \n",
    "\n",
    "1) Using CURL or any client like PostMan\n",
    "\n",
    "2) Using the Python SeldonClient\n",
    "\n",
    "### Using CURL from the terminal\n",
    "When using CURL, the only thing we need to provide is the data in JSON format, as well as the url, which is of the format:\n",
    "\n",
    "```\n",
    "http://<ENDPOINT>/seldon/kubeflow/<PIPELINE_NAME>/api/v0.1/predictions\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"meta\": {\n",
      "    \"puid\": \"k89krp6t7tfgb386nt6vc3iftk\",\n",
      "    \"tags\": {\n",
      "    },\n",
      "    \"routing\": {\n",
      "      \"cleantext\": -1,\n",
      "      \"tfidfvectorizer\": -1,\n",
      "      \"spacytokenizer\": -1\n",
      "    },\n",
      "    \"requestPath\": {\n",
      "      \"cleantext\": \"clean_text_transformer:0.1\",\n",
      "      \"tfidfvectorizer\": \"tfidf_vectorizer:0.1\",\n",
      "      \"lrclassifier\": \"lr_text_classifier:0.1\",\n",
      "      \"spacytokenizer\": \"spacy_tokenizer:0.1\"\n",
      "    },\n",
      "    \"metrics\": []\n",
      "  },\n",
      "  \"data\": {\n",
      "    \"names\": [\"t:0\", \"t:1\"],\n",
      "    \"ndarray\": [[0.6729318752883149, 0.3270681247116851]]\n",
      "  }\n",
      "}"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100    72    0     0  100    72      0    354 --:--:-- --:--:-- --:--:--   356\r",
      "100   599  100   527  100    72    516     70  0:00:01  0:00:01 --:--:--   588\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -X POST -H 'Content-Type: application/json' \\\n",
    "    -d \"{'data': {'names': ['text'], 'ndarray': ['Hello world this is a test']}}\" \\\n",
    "    http://127.0.0.1/seldon/kubeflow/$(kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}')/api/v0.1/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the SeldonClient\n",
    "We can also use the Python SeldonClient to interact with the pipeline we just deployed "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "data {\n",
      "  names: \"text\"\n",
      "  ndarray {\n",
      "    values {\n",
      "      string_value: \"Hello world this is a test\"\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "meta {\n",
      "  puid: \"qtdca40d3s0463nn4ginhkvc6t\"\n",
      "  routing {\n",
      "    key: \"cleantext\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"spacytokenizer\"\n",
      "    value: -1\n",
      "  }\n",
      "  routing {\n",
      "    key: \"tfidfvectorizer\"\n",
      "    value: -1\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"cleantext\"\n",
      "    value: \"clean_text_transformer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"lrclassifier\"\n",
      "    value: \"lr_text_classifier:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"spacytokenizer\"\n",
      "    value: \"spacy_tokenizer:0.1\"\n",
      "  }\n",
      "  requestPath {\n",
      "    key: \"tfidfvectorizer\"\n",
      "    value: \"tfidf_vectorizer:0.1\"\n",
      "  }\n",
      "}\n",
      "data {\n",
      "  names: \"t:0\"\n",
      "  names: \"t:1\"\n",
      "  ndarray {\n",
      "    values {\n",
      "      list_value {\n",
      "        values {\n",
      "          number_value: 0.6729318752883149\n",
      "        }\n",
      "        values {\n",
      "          number_value: 0.3270681247116851\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "import numpy as np\n",
    "import subprocess\n",
    "\n",
    "host = \"localhost\"\n",
    "port = \"80\" # Make sure you use the port above\n",
    "batch = np.array([\"Hello world this is a test\"])\n",
    "payload_type = \"ndarray\"\n",
    "# Get the deployment name\n",
    "deployment_name = subprocess.getoutput(\"kubectl -n kubeflow get seldondeployment -o jsonpath='{.items[0].metadata.name}'\")\n",
    "transport=\"rest\"\n",
    "namespace=\"kubeflow\"\n",
    "\n",
    "sc = SeldonClient(\n",
    "    gateway=\"ambassador\", \n",
    "    ambassador_endpoint=host + \":\" + port,\n",
    "    namespace=namespace)\n",
    "\n",
    "client_prediction = sc.predict(\n",
    "    data=batch, \n",
    "    deployment_name=deployment_name,\n",
    "    names=[\"text\"],\n",
    "    payload_type=payload_type,\n",
    "    transport=\"rest\")\n",
    "\n",
    "print(client_prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6) Visualise Seldon's Production ML Pipelines\n",
    "We can visualise the performance using the SeldonAnalytics package, which we can deploy using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts --namespace kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In my case, similar to what I did with Ambassador, I need to make sure the the service is a LoadBalancer instead of a NodePort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "service/grafana-prom patched\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl patch svc grafana-prom --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"LoadBalancer\"}]' -n kubeflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE\r\n",
      "grafana-prom   LoadBalancer   10.98.248.223   localhost     80:32445/TCP   64m\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl get svc grafana-prom -n kubeflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can access it at the port provided, in my case it is http://localhost:32445/d/3swM2iGWz/prediction-analytics?refresh=5s&orgId=1\n",
    "\n",
    "(initial username is admin and password is password, which will be requested to be changed on the first login)\n",
    "\n",
    "Generate a bunch of requests and visualise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    client_prediction = sc.predict(\n",
    "        data=batch, \n",
    "        deployment_name=deployment_name,\n",
    "        names=[\"text\"],\n",
    "        payload_type=payload_type,\n",
    "        transport=\"rest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You now have a full end-to-end training and production NLP pipeline 😎 \n",
    "![](img/seldon-analytics.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:reddit-classification] *",
   "language": "python",
   "name": "conda-env-reddit-classification-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
